A Restless Bandit Formulation of Multi-channel Opportunistic Access: Indexablity and Index Policy

We focus on an opportunistic communication system consisting of multiple independent channels with time-varying states. With limited sensing, a user can only sense and access a subset of channels and accrue rewards determined by the states of the sensed channels. We formulate the problem of optimal sequential channel selection as a restless multi-armed bandit process, for which a powerful index policy—Whittle’s index policy—can be implemented based on the indexability of the system. Exploiting the underlying structure of the multi-channel opportunistic access problem, we establish the indexability and obtain Whittle’s index in closed-form for both discounted reward and average reward criteria. These results lead to the direct implementation of Whittle’s index policy with remarkably low complexity. Furthermore, we develop a simple approach to evaluate the optimal performance under a relaxed constraint on sensing actions, which provides an upper bound of the optimal performance of the original restless multi-armed bandit process. The tightness of the upper bound and the near-optimal performance of Whittle’s index policy are illustrated with simulation examples. When channels are stochastically identical, we show that Whittle’s index policy is equivalent to the myopic policy, which has a simple and robust structure. Based on this structure, we establish approximation factors of the performance by Whittle’s index policy for stochastically identical channels. Index Terms Opportunistic access, dynamic channel selection, restless multi-armed bandit, Whittle’s index, indexability, myopic policy This work was supported by the Army Research Laboratory CTA on Communication and Networks under Grant DAAD19-012-0011 and by the National Science Foundation under Grants ECS-0622200 and CCF-0830685. Part of this work was presented at the 5th IEEE Conference on Sensor, Mesh and Ad Hoc Communications and Networks (SECON) Workshops (June, 2008) and will be presented at IEEE Asilomar Conference on Signals, Systems, and Computers (October, 2008).